OffsetManager: match upstream mark/next behaviour #713
Conversation
|
@eapache does this mean that all consumers that use the offset manager that upgrade past this patch will drop a single message during the upgrade? |
|
It depends on what you were using before. If you were sharing the offset topic with an upstream java If you were using the previous Sarama behaviour as documented (storing the offset of the last-consumed message) then an upgrade should cause you to consume a single message twice (the previous code will consume X and mark X, then the new code will read X, consume X, and mark X+1). Your consumer should already be able to handle that safely because it can occur in other scenarios. Note that either way you should ensure that code using the behaviour in this PR commits |
Upstream requires you mark last-consumed+1 and then returns that value directly. We were requiring you mark last-consumed and then adding one to the returned value. Match upstream's behaviour so that our offset tracking is interoperable. Fixes #705.
eapache
force-pushed
the
match-upstream-offset-manager
branch
from
b096b15 to
ccab789
Compare
|
I've added a more explicit warning to the changelog, but since consumers should already be able to handle duplicates (and this change will at worst cause a single duplicate message when a consumer restarts) I'm not too worried. |
|
cc @dim FYI - I don't think sarama-cluster is using our offset-manager but I know you took some inspiration from it so you might have the same issue. |
|
@eapache ok, that seems fine |
|
@eapache thanks for the heads up - fixed in bsm/sarama-cluster@6fdf904 |
Upstream requires you mark last-consumed+1 and then returns that value directly.
We were requiring you mark last-consumed and then adding one to the returned
value.
Match upstream's behaviour so that our offset tracking is interoperable.
Fixes #705.
@Kane-Sendgrid